The present invention relates generally to computer software, and specifically to programs enabling capture, storage and communication of the contents of computer displays.
There are a variety of computer screen capture tools known in the art. These tools enable the contents and appearance of a computer screen to be captured, or recorded, more or less in real-time. Generally, a sequence of screens is captured and is then stored to disk and/or transferred to another computer. Screen capture tools of this sort are useful, for example, in educational applications and in training and promotional demonstrations. Screen capture is also used by computer remote control tools.
Screen capture products for education, training and promotion include Lotus ScreenCam (http://www.lotus.com/ screencam), Hyperionics HyperCam (http://www.hyperionics. com) and TechSmith SnagIt/32 (http://www.techsmith.com) These products enable a user to record the contents of a computer screen to a file, while the computer is carrying out another program, and then to reproduce the recorded screen content from the file. They evidently work by encoding a bitmap image of the entire contents of the screen. Multiple screens in sequence may be recorded by encoding the differences between successive screens. This approach usually generates large amounts of processed data and very large output files. As a result, users may be limited to working at very slow refresh rates, on the order of one or a few frames per second, if they wish to record a full, active computer screen. Transferring the output files over a low-bandwidth computer network may be even slower. The alternative is to compromise on the content of the recording, typically by reducing the color resolution, by recording only a portion of the screen, or by simplifying the screen contents, by reducing the number of windows that are open on screen, for example.
OPTX International ScreenWatch (http://www. screenwatch.com) uses an alternative approach of capturing data sent to the computer""s display driver, in this case a proprietary driver developed for this purpose by OPTX. The display driver runs on a Microsoft Windows NT computer, which conveys the data to a separate client computer for recording. The data are stored in a proprietary format, which can subsequently be played back using a dedicated player program. The alternative approach employed by ScreenWatch enables faster, more efficient screen capture, but is limited to the complex, proprietary operating environment for which it was designed.
Computer remote control tools include Symantec PCAnywhere (http://www.symantec.com/region/can/eng/ product/pcanywhere) and LapLink.com LapLink (http://www.travsoft.com/products/llpro). These products enable a remote user to control a host computer and observe the screen contents of the host. They are not capable of keeping up with large or rapid changes on the host computer screen, even at high transmission bit rates between the host and remote computers.
It is an object of the present invention to provide improved tools for computer screen capture and playback.
It is a further object of some aspects of the present invention to reduce the volume of data required to capture complex computer screen contents.
It is still a further object of some aspects of the present invention to increase the frame rate at which computer screen contents can be captured.
It is yet another object of some aspects of the present invention to provide tools for computer screen capture that are applicable to a wide range of platforms and can be played back by a platform-independent player.
In preferred embodiments of the present invention, a computer screen capture tool treats windows on the computer display as objects, and records changes in the characteristics of these objects and relations among them from frame to frame. Preferably, a group of typical transformations of the windows is defined, and these typical transformations are encoded and recorded separately from other changes in the window contents. Generally speaking, the typical transformations are defined by an operating system of the computer, and they are therefore common to windows running different applications and can be encoded very compactly for all of the windows on the screen. In this way, the amount of bitmap data that must be recorded is substantially reduced relative to screen capture tools known in the art.
Displays with multiple active windows can thus be recorded and transmitted in real time, with high temporal resolution (i.e., high frame refresh rates), as well as full color definition and detail, while the computer is carrying out application tasks. No special resources are needed, in contrast to products known in the art such as the above-mentioned ScreenWatch. Different encoding schemes can be applied to the contents of different windows, depending on the type of contents (for example, video as opposed to text). In some preferred embodiments, movements of other on-screen objects, such as a mouse-driven cursor and other icons, are also encoded using typical transformations.
To play back the recorded screens, the windows and other objects are reconstructed in each successive frame by applying the encoded typical transformations to the windows and objects in the preceding frame. The window contents are then reconstructed inside the windows. Preferably, a screen player program for reconstructing the recorded screens is independent of the operating system of the recording computer. Most preferably, the screen player is provided in a platform-independent form, for example, in the Java language.
Preferred embodiments of the present invention are useful in a range of applications, including demonstrations and presentations, education and remote control, as described in the Background of the Invention.
There is therefore provided, in accordance with a preferred embodiment of the present invention, a method for capture of computer screens in a sequence of frames, including:
identifying a first set of one or more windows appearing in a first frame in the sequence, each window in the set having respective first-frame window characteristics and window contents;
encoding a description of the first set of windows indicative of the appearance of the computer screen in the first frame;
identifying in a second frame in the sequence a second set of one or more windows having respective second-frame window characteristics and window contents, the second set including one or more windows corresponding respectively to one or more of the windows in the first set;
determining one or more transformations applied to the first-frame window characteristics of the windows in the first set to generate the second-frame window characteristics of the corresponding windows in the second set; and
encoding a description of the second set of windows including the determined transformations, for use in reconstructing the computer screen as it appeared in the second frame.
Preferably, identifying the first set of windows includes identifying windows generated in accordance with an operating system of the computer, which associates each window with a respective function of the computer, such that the contents of the windows are determined by the respective functions. Most preferably, the respective functions include applications running under the operating system. Further preferably, determining the transformations includes defining transformations applied by the operating system, and which are applicable to different windows associated with different functions, generally irrespectively of the functions. Preferably, identifying the second set of windows includes querying the operating system regarding the characteristics of the windows. Additionally or alternatively, identifying the second set of windows includes intercepting events generated by the operating system.
In a preferred embodiment, identifying the first set of windows includes processing an image of the screen to identify the windows.
Preferably, determining the transformations includes defining a set of typical transformations, which are applicable to alter the window characteristics of the one or more windows, generally irrespectively of the contents of the windows. In preferred embodiments, the typical transformations are selected from a group of transformations including moving and resizing a window; minimizing, restoring and maximizing the size of a window; changing a Z-order of the windows, according to which two or more of the windows are overlaid one upon another on the screen; and scrolling and panning the window contents.
Preferably, encoding the description of the first set of windows includes encoding the first-frame window characteristics and the respective contents of the windows in the first set, and encoding the description of the second set of windows includes encoding the determined transformations and encoding changes in the contents of the windows in the second set with respect to the contents of the corresponding windows in the first set. In a preferred embodiment, encoding the changes in the contents of the windows includes applying different encoding schemes to the contents of different ones of the windows, wherein applying the different encoding schemes includes applying a video compression scheme to the contents of at least one of the windows, and embedding resultant compressed video data in the encoded description of the windows. Additionally or alternatively, applying the different encoding schemes includes applying different levels of encoding resolution to different ones of the windows in the second set.
Preferably, identifying the first and second sets of windows includes identifying windows associated with respective functions of the computer, wherein the identified windows include one or more user interface windows generated inside other identified windows for the purpose of controlling the functions associated therewith.
In a preferred embodiment, the method includes identifying first and second sets of one or more icons in the first and second frames, respectively, and determining transformations applied to the icons in the first frame to generate the icons in the second frame, to be encoded along with the description of the second set of windows. Preferably, the first and second sets of icons include a cursor.
In another preferred embodiment, encoding the first and second descriptions includes transferring the encoded descriptions over a communication link to a recipient computer. Most preferably, encoding the descriptions includes encoding the descriptions in a platform-independent format.
Alternatively or additionally, encoding the first and second descriptions includes storing the encoded descriptions in a memory.
There is also provided, in accordance with a preferred embodiment of the present invention, a method for reconstructing captured computer screens, including:
receiving an encoded description of a first set of one or more windows, having first-frame characteristics and window contents, which appeared on the computer screen in a first captured frame;
receiving an encoded description of a second set of one or more windows, having second-frame characteristics and window contents, which appeared on the computer screen in a second captured frame, subsequent to the first frame, the description of the second set of windows including a description of one or more transformations applied to the first-frame characteristics of at least one of the windows in the first set to derive the second-frame characteristics of a corresponding window in the second set; and
reconstructing the second captured frame responsive to the encoded descriptions of the first and second sets of windows.
Preferably, reconstructing the second captured frame includes decoding the encoded description of the first set of windows to determine the first-frame characteristics thereof, and applying the one or more transformations described in the description of the second set of windows to transform the first-frame characteristics into the second-frame characteristics of the at least one corresponding window. Most preferably, the encoded description of the second set of windows further includes encoded changes in the contents of the windows in the second set with respect to the contents of the at least one corresponding window in the first set, and reconstructing the second captured frame includes reconstructing the contents of the windows in the second set responsive to the encoded changes. In a preferred embodiment, the encoded description of the second set of windows includes compressed video data in a standard media format, and reconstructing the second captured frame includes invoking a standard media player to reconstruct video images in one of the windows.
Preferably, reconstructing the second captured frame includes reconstructing the first and second sets of windows substantially independently of an operating system under which the windows were generated, wherein reconstructing the first and second sets of windows includes operating a platform-independent screen player.
There is additionally provided, in accordance with a preferred embodiment of the present invention, apparatus for capture of computer screens in a sequence of frames, including:
a display; and
a processor, which is adapted to identify a first set of one or more windows appearing on the display in a first frame in the sequence, each window in the set having respective first-frame window characteristics and window contents, and to encode a description of the first set of windows, indicative of the appearance of the computer screen in the first frame, and
to identify, in a second frame in the sequence a second set of one or more windows appearing on the display, having respective second-frame window characteristics and window contents, the second set including one or more windows corresponding respectively to one or more of the windows in the first set, and to determine one or more transformations applied to the first-frame window characteristics of the windows in the first set to generate the second-frame window characteristics of the corresponding windows in the second set, and
to encode a description of the second set of windows including the determined transformations, for use in reconstructing the computer screen as it appeared in the second frame.
In a preferred embodiment, the processor is adapted to be coupled via a communication link to transfer the encoded descriptions to a recipient computer, which reconstructs the second frame responsive to the encoded descriptions of the first and second sets of windows.
In another preferred embodiment, the processor includes a memory adapted to store the encoded first and second descriptions.
There is further provided, in accordance with a preferred embodiment of the present invention, apparatus for reconstructing captured computer screens, including:
a processor, which is adapted to receive an encoded description of a first set of one or more windows, having first-frame characteristics and window contents, which appeared on the computer screen in a first captured frame, and
to receive an encoded description of a second set of one or more windows, having second-frame characteristics and window contents, which appeared on the computer screen in a second captured frame, subsequent to the first frame, the description of the second set of windows including a description of one or more transformations applied to the first-frame characteristics of at least one of the windows in the first set to derive the second-frame characteristics of a corresponding window in the second set, and
to reconstruct the first and second captured frames responsive to the encoded descriptions of the first and second sets of windows; and
a display, which is adapted to be driven by the processor to display the reconstructed first and second frames.
There is moreover provided, in accordance with a preferred embodiment of the present invention, a computer software product for capture of computer screens in a sequence of frames, the product including computer-readable media in which program instructions are stored, which instructions, when read by a computer, cause the computer:
to identify a first set of one or more windows appearing in a first frame in the sequence, each window in the set having respective first-frame window characteristics and window contents and to encode a description of the first set of windows, indicative of the appearance of the computer screen in the first frame, and
to identify in a second frame in the sequence a second set of one or more windows having respective second-frame window characteristics and window contents, the second set including one or more windows corresponding respectively to one or more of the windows in the first set and to determine one or more transformations applied to the first-frame window characteristics of the windows in the first set to generate the second-frame window characteristics of the corresponding windows in the second set, and
to encode a description of the second set of windows including the determined transformations, for use in reconstructing the computer screen as it appeared in the second frame.
There is furthermore provided, in accordance with a preferred embodiment of the present invention, a computer software product for reconstructing captured computer screens, the product including computer-readable media to be read by a computer that receives an encoded description of a first set of one or more windows belonging to a first captured frame, the windows having first-frame characteristics and window contents, and an encoded description of a second set of one or more windows belonging to a second captured frame, subsequent to the first frame, the windows having second-frame characteristics and window contents, the description of the second set of windows including a description of one or more transformations applied to the first-frame characteristics of at least one of the windows in the first set to derive the second-frame characteristics of a corresponding window in the second set, wherein program instructions are stored in the computer-readable media, which instructions, when read by the computer, cause the computer to reconstruct the second captured frame responsive to the encoded descriptions of the first and second sets of windows.